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Abstract— Computer vision is a multidisciplinary field in computational intelligence and artificial 
intelligence that guide intelligent systems and machine towards understanding the content of images or 
video. It has come under the spotlight in recent time due to the remarkable advancement and breakthrough 
day-by-day such as autonomous driving, intelligent systems, pedestrian system, robotics, medical imaging, 
remote sensing, object detection, security, speech sequence, image registration, biometric technologies, 
image retrieval and video processing to mention just a few. The aim of this work is provide a systematic 
review in the application of deep learning in various aspect of computer vision that is object detection, 
object classification, scene classification, image segmentation, image retrieval, image registration, object 
recognition, feature extraction, image fusion and anomaly detection by providing detail description in a 
systematic way of all the deep learning techniques; Convolutional Neural Network (CNN), Deep Boltzmann 
Machine (DBM) and Deep Belief Network (DBN) that are applied in solving this problems of computer 
vision. This work also explored all the review and survey researches that applied deep learning techniques 
in computer vision in a systematic format and identify the countries that the researches was conducted 
which is one of the unique attribute of this paper that make it exceptional in a state-of-art review and 
surveys. In addition to datasets that used in the recent research of computer vision with the software and 
toolboxes used for the implementation of deep learning techniques in computer vision. We have also 
identified all the top journals (ISPRS Journal of Photogrammetry and Remote sensing, IEEE Geoscience and 
Remote Sensing Letter, IEE Transaction of Geoscience and Remote sensing etc.) and conferences 
proceedings (IEEE Conference of Computer vision and Pattern recognition) for advanced publication of 
computer vision researches in the world. This work would serve as a roadmap for the any researcher that 


want to use deep learning algorithms in computer vision. 


Keywords— Computer vision, Deep learning, Convolutional Neural network, Deep Boltzman Machine, 


Deep Belief Network, object detection. 


BACKGROUND 

Deep Learning algorithms has achieved exponential progress and remarkable performances in computer 
vision. Sequel to it promising performances it is widely applied in several areas of computer vision these 
include; image classification object detection scene classification and land use and land cover Chen, Lin, 
Zhao, Wang, Gu (2014); Zou, Ni, Zhang, Wang, (2015); Chen, Zhao and Jia, (2015); Romero, Gatta, Camps,- 
valls, (2016); Cheng and Han, (2016); Marmanis, Datcu, Esch and Stilla, (2016); Yu et al., (2017); Kussul, 
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Skakum, Shelestov, (2017); Sharma, Liu, Yang and Shi, (2017); Vetrivel, Gerke, kerl, Nex and Vosselman, 
2018). Specifically for image classification Krizhevsky, Sutskever, and Hinton, (2012); object detection Lin, 
Dollar, Girshick, He, Hariharan, and Belongie, (2017); Dai, Li, He, and Sun, (2016); Zhou, Ni, Geng, Hu, and 
Xu, (2018); Zhang, Wen, Bian, Lei, and Li, (2018); Singh and Davis, (2018). Motion detection, object 
positions, image segmentation, event tracking, pattern recognition and texture extraction Godfellow, 
Bengio and Courvile, (2013). Speech recognition, Zhang, Pezeshki, Brakel, Zhang, Bengio, and Courville, 
(2017). Moreover, recent reviews are also devoted in the application of deep learning to remote sensing 
Liu Y., Chen, Wang, Ward and Wang, (2018) but they have focused specifically for data fusion and left others 
areas. In addition to Zhu, Tuia, Mou, Xia, Zhang, Xu and Fraundorfer, (2017) focused on 3D modeling 
application but they ignore the very important aspect of remote sensing such as classification and object 
detection. But Lei, Yu, Xueliang, Yuanxin, Gaofei and Brian, (2019) conducted a thorough review on the 
application of deep learning in remote sensing with a clear analysis as well. We have reviewed many 
articles, survey researches and conferences in computer vision however, most of the scholars prefer to use 
standard datasets in training and testing the performance of their system as in (Xia et al., 2017) Uc-Merced 
dataset also used in (Zhao and Duo, 2016) (Scott et al, 2017) while (Zhong et al, 2016) and (Han et al., 
2017). Used WHU-RS dataset and also (Zou et al., 2015) and (Yang et al., 2017) used RSSCN7 dataset for 
their training and testing their models. In addition to some scholars had extend the training samples 
because of the nature of the dataset to enhance the performance of the system (Marmani et al., 2016) and 


(Scott et al., 2017). 


This paper reviews the recent research articles and peer reviewed papers from top journals in computer 
vision. We have subjected our search to JEEE and Elsevier Journals with some few Q1 journals in the 
computer vision journals based on this search we have found (687) research articles and (120) reviews, 
survey, comprehensive survey, analysis and overview researches. All this articles are mostly found in (32) 


Journals including conference proceedings. 


However, we have not utilized all the conference papers obtained because they do not have sound academic 
background so we have used very few. This reviewed is quite different from the previous reviews because 
we have explored extensively on both deep learning and computer vision techniques. We have shown the 
top journals in computer vision, we have also used systematic review format for easily understanding and 
identifying the gaps. We have also review the datasets used in computer vision researches and also the 


software for implementing computer vision researches. 


The major difference between this reviews and previous reviews is that it covers all aspect of computer 
vision such as object detection, image classification, land cover, image segmentation, scene classification, 
image retrieval, target recognition, semantic segmentation, feature extraction, image fusion and anomaly 
detection to mention just a few which was not found in the previous reviews. In the previous reviews they 


focus on a specific computer vision technique thus; segmentation, classification object detection and so on. 
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Table 1: List of Abbreviation used in this research 











Abbreviation Full meaning 
1. DL Deep Learning Convolutional Neural 
2. CNN network 
3. DBN Deep Belief Network 
4. DBM Deep Boltzmann Machine 
5. GAN Generative Adversarial Network 
6. AE Auto-encoder 
7. USL Unsupervised Learning 
8. NN Neural Network 
9. DSL Deep Supervised learning 
10. WSL Weekly Supervised Learning 
11. RGB Red Green Blue 








Deep learning algorithms 

Deep learning is a family of computational methods and methodologies, techniques and models 
encompassing many categories of neural networks (CNN, RCNN, DNN), hierachical probablistic models and 
a many unsupervised and unsupervised learning algorithm (Athanasioa et al., 2018) deep learning made 
unprescendented progress in computer vision such as image classsification segmentation, image retrival 
and scene classification. In the context of computer vision image classification or segmentation is the 
processing of classifying various objects in an image such as human being, mountains, water bodies, trees, 
forest, ship, car, and mineral resources etc. classsification and segmmentation are implemented using 
supervised, unsupervised and semisupervised learning algorithms. In all these algorithms used feature 
extraction is the major parameter used in image classification. As applied by researchers using CNN, SVM, 
PCA, K-means algorithm, minimun distance classifier and bayesian classifier in (Tui et al., 2015; Han et al., 
2018; Srivastava et al., 2019; Cao et al., 2020; Rasti 2020; Noh et al., 2015; Hu and Koltun, 2015; Riese, 
Keller and Hinz, 2020). In this section we have explored several classification of deep learning techniques 


applied in computer vision such as CNN, DNN, RNN, DBM and GAN. 


Deep learning classification 

Deep learning algorithms has achieved unprecedented progress in different application domain ranging 
from security, bioinformatics, social media, remote sensing, digital forensic, medical imaging and computer 
vision. Categorically deep learning is classified into; deep supervised learning, deep unsupervised learning, 
semi-supervised learning and reinforcement learning. Deep supervised learning algorithms are trained 
with a label dataset. Data most content inputs and the resulting outputs. It is the simplest technique to 
implement among different categories of DL techniques because it has the ability to generate output from 
the prior knowledge learned. There are several techniques in supervised learning; CNN, DNN and RNN 


(Laith et al., 2021). 
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Convolutional neural network is a deep learning technique that consist of inputs layer and many of feature 
detection layer which perform convolution, pooling and Rectified Linear Unit, at convolutional stage it 
applies convolutional filters to activate certain features in the image, at the pooling stage it reduces the 
number of pixel in the image by using non-linear down sampling or sub-sampling. Moreover, pooling 
process does not affect the depth dimension of the image, average pooling and max pooling are the most 
widely used strategies in this unit (Boureu, Ponce and Lecun, 2010). Many other pooling layer are also 
explored in the recent literature; spatial pyramid (He, Zhang, Ren and Sun, 2015) and (He, Zhang, Ren and 
Sun, 2014). And def-pooling (Ouyuan et al., 2015) stochastic pooling in (Wu and Gu, 2015) and at the 
rectification linear unit it lets negative values to zero and maintain all the positive values. The second to the 
last layer is called classification layer, it converts 2D feature map to 1D feature map vector, it is fully 
connected with N-dimension output where N is the number of objects to be classified and the final output 


will show the probability that the input image belong to the class of object under review. The architecture 


of CNN utilizes three major techniques local reception fields, tied weights and spatial sub-sampling 


(Athanasios et al., 2018). 
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Fig. 2.1. CNN architecture (Sumita, 2018) 


Deep unsupervised learning these category of deep learning requires no label data for training it learned 
from the significant feature of the data and relationship in the data but the major disadvantage of this 
technique is the inability to provide accurate concerning data sorting as well as the computational 
complexity of the dataset. Several techniques are available in this categories; Deep Boltzmann Machine 
(DBM) and auto encoder which are widely applied in clustering. Deep Boltzmann machine all the 
connections in this technique are undirected with multiple hidden layers it applies stochastic maximum 
likelihood during the training process and require no label in the training dataset. The advantage of this 
technique is the ability to optimize the parameters of all layers (Srivastava and Salakhutdinov, 2014). Deep 
belief network in this technique two top layers form undirected graph and the lower layer form directed 
generative model. DBN are trained using greedy method as proposed in (Hinton and Salakhutdinov, 2006). 


Deep semi-supervised learning requires semi-labeled dataset for training. They are widely applied for text 
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document classification. This includes; generative adversarial network (GAN) the disadvantage of this 


technique is that unwanted features are learned from the input data and make wrong decision. 


Deep learning in object detection 

In this section we conducted a thorough survey on the application of deep learning in computer vision such 
as object detection in images, motion detection in videos and pattern recognition. An object detection is the 
process of detecting objects in a digital image such as human settlement, mountains, forest, cars, roads, etc. 
(Ouyandng, Zeng and Wang, 2017). The taxonomy of object detection includes; edge detection, salient 
object detection, pose detection, face detection highlight detection, fine grained visual detection, multi- 
domain object detection, point cloud 3D object detection, 2D 3D pose detection and pedestrian detection. 
Basically deep learning based object detection in computer vision is divided into two categories; one stage 
object detectors and two stage object detectors. In this categories many architectures are recently 
introduced and applied in object detection. One stage objectors are commonly applied in object detection 
by using region of interest. YOLO (you only look once) it was introduced in 2015 it normally cluster the 
input images into patches and locate the object in all the patches, SSD it was introduced in 2016 in order to 
overcome the major shortcomings identify in YOLO, RetinaNet which was introduced in 2017 by Tsing et 
al., (2017). Two stage object detectors; the most common architectures used under this category that are 
applied in object detection are R-CNN which was introduced by (Ross et al., 2014) however, it is time 
consuming in training and testing. SPP-net was introduced in 2014 and proposed by (Kaiming et al., 2015). 
Fast-CNN is an architecture that combine the future of R-CNN and SPP-net, Faster R-CNN was introduced 
by (Shaoging et al., 2015) it uses a fully connected CNN called Region proposal network (RPN), R-FCN it 
was introduced in 2016 by Jifeng et al., (2016) its apply region based convolutional neural network and 


Mask R-CNN was introduced by (Kaiming et al., 2017). 


Table 2: Deep Learning Algorithms in Computer vision 

















Authors and date Techniques Focus 

Goodin, Anibas and Bezymennyi (2015) CNN (Object based classification in 
Wang, Song, Chen, Yang, (2015). CNN Road network extraction 
Arshitha and Biju (2020). CNN Detection in satellite image 

Scott et al ., (2017) CNN Land cover classification 

Luus et al., (2015) CNN Land used classification 

Kadhim and mohammed (2020) CNN Satellite image classification 
Imamoglu et al., (2018). CNN Multi-spectralimage classification 
Li, Zhang, Huang and Yuille (2018) DRCN Deep network 

Mou and Zhu (2017) CNN Geo-spatial object detection 
Deepthi, Sandeep and Suresh (2021) CNN Hyper spectral image 

Zhong et al., (2016) CNN Classification of object in satellite 
Deng et al., (2017) CNN Building and road extraction 
Liang and Xiaolin, (2015) RCNN (Object detection in remote sensing 
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Liang and Hu, (2015) 

Gao et al., (2017) 

Li et al., (2016) 

Cheng, et al., (2016) 

Cheng, et al., (2018) 

Zhang, Du and Zhang, (2016) 
Boulleg and Farah, (2018) 
Zhou, Newsam, Li and Shao (2017) 
Ye et al., (2018) 

Zhou, Deng and Shao, (2018) 
Cao, Ni and Dou, (2016) 

Zhang et al., (2020) 

Zhang et al., (2018) 

Yu et al., (2017) 

Mahdianpari et al., (2018) 
Alshehhi et al., (2017) 
Bergado et al., (2016) 

Kussul et al., (2017) 
Castellucio et al., (2015) 

Sevo and Avramovic, (2016) 
Volpi and Tuia, (2017) 

Rene, He, Girshick and Sun (2017) 
Muhammad et al., (2018) 
Zhang, Shi and wu (2015) 
Salberg, (2015) 

Sevo and avramoric, (2016) 
Zhang, Zhang and Xu (2016) 
Tang et al (2015) 

Han et al., (2015) 

Zhang, Du, and Zhang (2016) 
Hu,Xia,Hu, and zhang (2015) 
Yang, yin, and Xia (2015) 

Hu et al (2015) 

Cheng, Zhou, and Han (2016) 
Zou, Ni, Zhan and Wang (2016) 
Geng, fan , wang, ma, li, and chen (2015) 
Geng, wang, fan, and ma, (2017) 
Hou, Kou, and Jiao (2016 
Zhang, Ma, and zhang, (2016) 
Qin, Guo and sun, (2017) 

Zhoa, Jiao, Gu, ana) eu) 


RCNN 
DCNN 
DL 
CNN 
CNN 
CNN 
CNN 
CNN 
CNN 
RCNN 
OCNN 
OCNN 
CNN 


| DCNN 
CNN 








DL 
CNN 
CNN 
CNN 
CNN 
RCNN 
CNN 

DL 
DCNN 

NN 

SL 


| DNN 


WSL 
DNN 
DCNN 
DL 
USL 
CNN 
DL 
DCA 
DSNN 
MLE 
AE 
BM 
DBN 
CNN 


Object recognition 

‘Object Recognition 

‘Image classification 

Image classification 

Scene classification 

‘Scene classification 

Scene classification 

Scene classification 

Image retrieval 

Image retrieval 

Object detection 

Land cover classification 

Urban land used classification 
Intelligent land slide detection 
Image classification and segmentation 
Road extraction and building 
Image classification 
Classification of land cover 
Land used classification 

Object detection in aerial images 
Semantic labeling of images 
Real time object detection 

Early fire detection 

Oil and tank detector 

Detection of seal in satellite images 
Object detection in aerial images 


Aircraft detection 





Ship detection 
Object detection 
Scene classification 
‘Scene Classification 
Image classification 
‘Spectral clustering 
‘Object detection 
‘Scene classification 
Scene classification 
Image classification 
Image classification 
Image classification 
Object Classification 


Image classification 
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Duan Liu, Jiao, Zhao, (2017) 

Ding, chen, Liu, and Huang, (2016) 

Chen et al., (2016) 

Chen and wang (2014) 

Li, wu, and Du (2017) 

Mou, Ghamisi and Zhu(2017) 

Santaraetal (2017) 

Makantasis et al (2015) 

Hu, Huang, wei, zhang, and li (2015) 

Tao, pan, Li and zou(2015) 

Chen, Zhao and Zou(2015) 

Chen, lin, Zhao, Wang, and Gu (2017) 
Ghamisi, Chen, and Zhu, (2016) 

He, Zhang, Ren and Sun, (2016) 

Girshick, Donahue, Darrell and Malik, (2016) 
Long, Shelhamer and Darrell, (2015) 

Noh, Hong and Han, (2015) 

Chen, Jiang, Li, and Ghamisi (2016) 

| Kampftmeyer, SAlberg and Jenssen (2016) 





Zhong, Yang, Huang, Zhong and chen (2016) 





CNN 
DCNN 
DL 
DL 
DRNN 
DL 
DSL 
DCNN 
USL 

DBN 
DBN 
DL 
CNN 
DL 
CNN 
CNN 
CNN 
CNN 
DCNN 


_ CNN 





[Image segmentation 


Target recognition 
Target recognition 
Target recognition 
Anomaly detection 
Image classification 
Image classification 
Image classification 
Image classification 
Image classification 
Spectral classification 
Image classification 
Image classification 
Image recognition 
Object detection 
Semantic segmentation 
Semantic segmentation 


Feature extraction & classification 


Semantic segmentation 





Image fusion 





Survey Researches that applied deep learning techniques in computer vision 


In this section we have subjected our search to only survey and review researches that recently applied 


deep learning techniques in computer vision. Moreover, we have strictly also concentered on where the 


research has conducted. To the base of our knowledge and findings in this work shows that higher 


percentage of computer vision researches are conducted in China while the remaining smaller percentage 


are shared among the rest. However, it comes to our noticed also that no one researches that was conducted 


in Africa in respect to computer vision. 


Table 3: Survey Researches 





Author and date Techniques Country Description 

Ma, et al, (2019) Deep learning China Mata analysis and review 
Zhu et al., (2017) Deep learning China Comprehensive review 
Tsagkatakis et al., (2019) Deep learning Survey 

Ball, Anderson and Chan., (2019) Deep learning China Comprehensive survey 
Song, Gao, Zhu And Ma (2019) CNN China Survey 

Hyedari and Mountrakis (2019) DNN Meta-analysis 

Zhang, Zhang And Du (2020) Deep learning China Technical tutorial 

Lu, Sun, and Zhang (2019) CNN China Feature aggregation 





Paoletti, Haut and Plaz (2019) 


Deep learning 


Review 
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Parikh, Patel and Patel, (2020) Deep learning Review 

Li etal, (2019) Deep learning China Overview 

Pashae, Kamangir, Starck And Tissot, (2020) Deep learning Review 

Li, Zhang, Xue, Jiang and Shen (2018) Dee learning China Survey 

Liu et al., (2018) Deep learning China Advances and Feature 
Rawat and Wang, (2017) DCNN India Comprehensive survey 
Xiao et al., (2017). DL China Comprehensive review 
Athanasios et al., (2018) DL Survey 

Xiangwei, Doyen and Steven (2019) DL China Recent advances 
Lincheng et al., (2019) DL China Survey 

Accoss-Garcia et al., (2018) DL Spain Analysis 

Zhao, Feng, Wu and Yan (2017) DL China Survey 

Payal et al., (2020) DL Survey 

Jiao et al., (2019) DL China Survey 

Abhishek et al., (2021) DL Canada Survey 

Vipul and Roohie, (2020) DL India Review 











Softwares for Compuetr Vision 

Many softwares are available for the implementation of deep learning algorithms in computer vision 
researches. In this work we review some of the most recently applied softwares for computer vision. 
MatconvNet (MATLAB), Tensorflow (C++ and python), R, Caffe (C++), Torch (C and Lua), Keras (python), 
Deeplearning4j (Java), MxNet, Theano (Python), Gluon (C++), OpeenDeep (Python), NTK (C++) and 


ConvnetJs. Some of these softwares are open source and some are not. (Jude and Vania, 2017). 


Datasets used in Computer Vision 

DL researches require huge amount of data to make correct decision. There are many datasets online for 
different application domain. In computer vision the most common once are; ImageNet with 1000 classes 
of images prepared for object detection and classification researches. MNIST are gray scale images with 10 
different classes of image prepared for hand written digit classification, Microsoft COCO with 80 different 
image classes prepared for object detection, Google Open images with 350 classes of sample images applied 
in image segmentation and object detection, Kinetics with 400 samples prepared for human action 
detection researches, YouTube8m with 8 million classes of videos for video classification. Caltech consist 
of 101/256 sample of images, CIFAR consists of 32x32 RGB images of various classes, COIL is an RGB 
natural image dataset used for object classification. In the area of remote sensing many datasets are also 
available and used by many experts in computer vision to train deep learning algorithm for image 
classification, object detection and scene classification. Zurich summer data set it consist of 20 images 
acquired in Zurich Switzerland in 2002 which can be obtained from (Volpi, 2017). ISPRS 2D semantic 
labeling challenge which consist of an average pixel values of 2,000 X 3,000 in which 17 tiles are labeled 
and other half are not labeled. These data set can be obtained from (ISPRS, 2017). Zeebruges dataset, this 
dataset consists of 10,000 X 10,000-pixel value in which all the image are labeled. (Xiao et al, 2017). UC 
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Merced dataset. This data set consist of aerial images with 256 X 256 pixel values which consist of 21 classes 
and each class is having 100 images. This dataset can be obtained from (NSf, 2010) there are used and 
applied for scene classification. Northwestern Polytechnic University Remote sensing image scene 
classification dataset this dataset consists of 31,500 aerial image with 45 different scene classes which are 
used and applied for scene classification and is obtainable from (EScience, 2016) SARPTICAL dataset this 
dataset is used in image registration and matching based research. (Xia et al, 2017). PASCAL VOC it contains 
20 categories with four (4) classes of vehicles, animals, people and household objects introduced in 2005 
used for object detection researches. VISDrone 2018 it was introduced in 2018 from China, consist of 
images and videos for object detection and tracking. ETH is a pedestrian datasets consist of 1804 frames 
on three classes of videos (Vipul and Roohie, 2020). INRIA is also a pedestrian detection datasets consist of 
2120 images in which 1832 images for training and 288 for testing (Vipul and Roohie, 2020). PASCAL FACE 
datasets consist of 1335 labeled faces from 851 different categories of images it is widely applied in facial 
recognition researches (Vipul and Roohie, 2020) and FDDB datasets is also facial detection datasets consist 


of 5171 faces classified into 2845 images. 


JOURNALS REVIEWED 
In this section we have subjected our search to only top jounals in computer vision from IEEE, Elservier, 
Science Direct, conference proceedings and other related journals which are almost thirty three journals 


including few conference were reviewed in this work as shown in the table below; 


Table 4: Journal reviewed 





NYA Title Publisher 

1. IEEE Geoscience and Remote sensing letter 
2. ISPRS Journal of photogrammetry and remote sensing IEEE 
3. JEEE international conference on mechatronic and Automation 

ISPRS 
4. Neutral Networks 

IEEE 
5. JEEE Conference on Computer Vision and Pattern Recognition 

ELSEVIER 
6. Advances in Neutral Information Processing System 

IEEE 
7. Arxiv preprint 

ELSEVIER 
8. Information fusion 

ARVXIV 
9. IEEE Geoscience and Remote Sensing Management 

ELSEVIER 
10. IEEE Transaction of Geoscience and Remote Sensing 

IEEE 
11. Internal Journal of Remote Sensing 

ELSEVIER 
12. Proceeding in Computer Science 

IEEE 
13. Applied science 

IJRS 
14. IEEE international Geoscience and Remote Sensing Symposium 

IEEE 
15. Remote Sensing 

ELSEVIER 
16. IEEE Geoscience and Remote Sensing Magazine 

IEEE 
17. Sensors 

ELSEVIER 
18. Journal of Applied Remote Sensing 
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19. Big Earth Data IEEE 
20. International Journal of Image and Data Fusion ELSEVIER 
21. Remote Sensing and Image Classification JARS 
22. Neutral Computation BED 
23. Computational intelligence and Neuron Hindawi IJIDF 
24. IEEE Transaction on Pattern Analysis and Machine Intelligence ELSEVIER 
25. Neurocomputing ELSEVIER 
26. Journal of bag data Springer HINDAWI 
27. Springer International Publishing IEEE 
28. Computer Vision ELSEVIER 
29. American Association for the Advancement of Science: Science ELSEVIER 
30. Journal of machine learning research SPRINGER 
31. IEEE Journal of Selected top application on Earth observation and remote sensing SPRINGER 
ELSEVIER 
SCIENCE 
JMLR 
IEEE 
CONCLUSION 


In this work we conducted a thorough reviw in the application of deep learning techniques in computer 


vision. We have subjected our serch towards various categories of deep learning algoriths together with 


their associated architectures that are applied in computer vision researches, we have used systematic 


format of literature reviw and explored critically the state-of the-art in all aspect of computer vision and 


also explored survey and review researches in the state-of-art. furthermore, we have summarized the 


major softwares and toolboxes use for the implementation of computer researches in addition to the 


datasets that are suitable for all aspect of researches in computer vision. 
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